WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCX 

INTERNATIONAL APPUCATION PUBUSHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification ^ : 
H04N 



A2 



(11) International Publication Number: WO 98/34397 

(43) International Publication Date: 6 August 1998 (06.08.98) 



(21) International AppUcaUon Number: PCT/US98/0I741 

(22) International Filing Date: 30 January 1998 (30.01.98) 



(30) Priority Data: 
60/036.655 
09/012,255 



30 Januaiy 1 997 (30.0 1 .97) US 
23 January 1 998 (23.0 1 .98) US 



(71) Applicant: APPLIED SCIENCE FICTION, INC. [US/US); 

Suite 500. 3925 West Braker Road, Austin, TX 78759-5321 
(US). 

(72) Inventor: EDGAR, Albeit, D.; 3912 Eton Lane, Austin, TX 

78727-6069 (US). 

(74) Agents: GLENN, Michael, A. et al.; Law Offices of Michael 
A. Glenn, P.O. Box 7831, Mcnlo Park, CA 94026 (US). 



(81) Designated States: AL. AU. BA. BB, BG, BR, CA. CN, CU, 
CZ, EE, GE. HU, ID. IL. IS. IP. KP, KR. LC. UC, LR. LT. 
LV, MG, MK, MN, MX, NO, NZ, PL. RO. SG. SI, SK, 
SL, TR, TT, UA. UZ, VN. YU, ARIPO patent (OH, GM, 
KE. LS, MW, SD, SZ, UG, ZW), Eurasian patent (AM, AZ, 
BY, KG, KZ. MD. RU. TJ, TM), European patent (AT, BE, 
CH. DE, DK, ES, FI. FR, GB, GR, IE, IT, LU, MC, NL, 
PT, SE), OAPI patent (BF, BJ, CF. CG, CI, CM, GA, GN. 
ML. MR. NE. SN. TD. TG), 



Published 

Without international search report and to be republished 
upon receipt of that report. 



(54) TiUe: FOUR COLOR TRILINEAR CCD SCANNING 
(57) Abstract 

A single pass scanner having a triUnear array, a source 
of white light, filters of the three primary colon and a separate 
source of infrared light is used in various methods of removing 
medium-based defects from a scanned film image. The method 
generates an infrared channel in addition to the common visible 
channels by covering the parallel rows of sensors in the trilinear 
array respectively with a red, green and blue filter to create the 
three color channels. Normally, each of the three color filters also 
passes infrared light, which is removed by filters external to the 
sensors. In a specific embodiment, interstitial in time between 
two visible light scans, the sensor is exposed to infrared light for 
a single scan. As the trilinear array sweeps across an image in 
time and spatial synchronization with the exposing lights, at least 
two visible channels and an infrared channel arc generated. 
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FOUR COLOR TRILINEAR CCD SCANNING 
RELATED APPLICATION 

5 

This application claims the benefit of U.S. Provisional Application No. 
60/036,655, filed January 30, 1997. 

FIELD OF THE INVENTION 

10 This invention relates to image enhancement and recovery, and more particularly 

to a method and apparatus for scanning color image data. 

BACKGROUND OF THE INVENTION 

Ever since the first image of an object was captured on film, a serious problem was 
15 apparent which has continued to plague the field of image capture and reproduction to the 
present day, namely imperfections in the recording medium itself which distort and 
obscure the original image sought to be captured. These imperfections occur in 
inmmierable forms including dust, scratches, fingerprints, smudges and the like. Archival 
polypropylene sleeves employed to protect negatives even contribute to the problem by 
20 leaving hairline surface scratches as the negatives are pulled out of and replaced into the 
sleeves. 

One method of approaching the problem of defective images is described by the 
present inventor in U.S. Patent No. 5,266,805 issued to Edgar. In that system, unage data 
is stored on a recording medium or substrate containing non-image imperfections such as 

25 fihn having surface scratches, wave smudges, bubbles, or the like, which give rise to 
undesirable artifacts in images subsequently retrieved fi-om the medium or substrate. 
Means are provided for deriving fi-om the medium separate images in the red, green, blue 
and infixed portions of the electromagnetic spectrum corresponding to the image stored. 
The mfrared image is used as an indicator or map of the spatial position of the non-image 

30 imperfections on and in the mediimi so the defects can be reduced or eliminated. In this 
way, the desired underlymg image is recovered. 

The fimdamentals of scanner technology for scanning and storing images in digital 
form are well developed and available in commercial products. Scanners receive an optical 
image, and divide it into many points, called pixels. The scanner measures light at each of 

35 these points to give each pixel a numerical value. Software associated with the scanner 
can then manipulate and store the pixel data. A color scanner measures several numerical 
values for each point, one for each primary color of red, green and blue. For example, a 
scaimer may measure a pink pixel to have 80% red, 40% green, and 40% blue. All 
numbers representing one of the primary colors are grouped by the software into a 
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channel, or single color image. Some scanners measure only one primary color at a time, 
then back up, change either the light source or a color filter to a second color, and measure 
only that second color on a second pass of the image. These are called multipass 
scanners. An example of a multipass scanner is the RFS 3570 made by Eastman Kodak 
5 Company. 

Other scanners make a single pass, and collect all color information in that one 
pass. One type of single pass color scanner uses a linear array, or line of sensors. This 
array is moved, or scanned, perpendicularly across the image to scan all points on the 
image line by line. During this scan a light source is rapidly switched between the various 

10 colors to be sensed. One complete cycle of the colored light sources occurs for each line 
of pixels in the image. Thus a single point on the image is seen by a single sensor in the 
anay first by one color, then by another color, and typically also by a third or fourth 
color, before that sensor moves on to another line of pixels on the image. Software is used 
to store the color-specific numerical information for each pixel into the appropriate 

1 5 channel so that at the end of the scan, multiple channels are available (one for each color). 
An example of a single pass scanner is the LSI 000 made by Nikon Corporation. 

Another type of single pass scanner illuminates the image being scanned with light 
containing all visible colors, but places tiny color filters over the sensor elements so at any 
point in time there are sensors receiving each different color. One such method positions 

20 three linear arrays side by side to form a "trilinear array." One of the three arrays is 
placed under a red filter, one under a green filter, and the third under a blue filter. 
Typically these colored filters pass infrared light in addition to the color they are intended 
to pass, and for this reason the infrared light must be removed firom the optical path by a 
separate fiker elsewhere in the scanner, or the scanner must use a light source containing 

25 no infrared light. An example of a scanner employing a trilinear array is the SprintScan 35 
made by Polaroid Corporation. 

A problem arises when applying media surface defect correction to single pass 
scanners using trilinear arrays. The filters on standard trilinear arrays distinguish red, 
green, and blue light, but not the fourth "color" of infrared light necessary to practice 

30 surface defect correction. Further, to be compatible with existing color dyes that pass 
infirared light, infrared light is often removed from the optical path prior to reaching the 
sensor, precluding addition of a fourth line of sensors sensitive to infirared light. 

The three sensor lines of a trilinear array typically are spaced by an integer 
multiple of the separation distance between pixels in the image. If the spacing integer is 

35 eight, then a specific point on the image may be sensed in red, and exactly eight steps later 
it may be sensed in green, and eight steps after that it may be sensed in blue. Thus, the 
same pixel is scanned at three different times. The software then realigns the color 
channels by moving each a multiple of eight steps so as to group together all of the pixels 
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sensed in the same color. The spacing between sensor lines of the array is usually a 
power of two (such as 2, 4, 8, 16, 32, etc.) multiplied by the pbcel spacing distance to 
allow the designer the option to choose submultiple resolutions. For example, if the 
spacing were 1 /250th of an inch, the scanner could operate at 2,000; 1,000; 500; or 250 
5 dots per inch, while retaining alignment between the colors with the appropriate offset 

Trilinear array scanners operate at very high speed because three lines of an image 
are scanned simultaneously. In addition, they provide very good color registration 
because of the single pass and in spite of generally low cost transport mechanics. Thus, 
there is no need to halt movement of the image at each scan line, and this further increases 
10 the speed of the scan. However, the market needs an image scanner with the speed and 
cost advantages of a single pass trilinear array which also includes surface defect 
correction capabilities. 

Yet another type of single pass scanner uses an area array to cover a two 
dimensional region of the image at once rather than mechanically scanning with a linear 
1 5 array to cover the region. One such scanner called a color filter matrix further incorporates 
tiny color filters over each element in the area array. In a specific implementation used in 
a Kodak digital camera, half the sensor elements lie under tiny green filters incorporated 
on the sensor array in a checkerboard pattern. The other half of the sensor elements in the 
checkerboard are behind alternating red and blue filters. Thus, a quarter of the sensors 
20 respond to red light, half respond to green light and a quarter respond to blue light. In 
another implementation in a Polaroid digital camera, an entire column of sensors is behind 
tiny red filters, the adjacent column of sensors is behind tiny green filters, and the next 
column of sensors is behind tiny blue filters. This pattern repeats in additional columns. 
In yet another implementation common in video cameras, the colored filters on even rows 
25 are green and magenta, and the filters on odd rows are cyan and yellow. Because the array 
is not mechanically scanned, each pixel in the image is measured with only one of the three 
colors, as opposed to the other scanners discussed so far that measure each pixel of the 
image with three colors. As with the trilinear array discussed above, all the colored filters 
pass mfrared light, and therefore infrared light must be removed by a separate filter 
30 elsewhere in the optical path. 

Another important consideration for image scanners is data compression due to 
the rather large amount of pixel data which may be detected by image scanners. Scanners 
that only sense a single specific color from each specific pbcel, such as those employing a 
color filter matrix, produce only one-third as much raw data as a scanner that senses all 
35 three colors from each pbcel, and therefore such scanners employ a form of data 
compression. For a purely black and white image, detail can be resolved at the full pbcel 
resolution of the scarmer because for a black and white image, it does not matter whether a 
pixel senses in red, green, or blue light. On the other hand, problems may occur in the 
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details of a color image where a point of white light aligns with a red sensor and so 
appears red, or with a blue sensor and so appears blue. It is known that this aliasing 
problem can be reduced by controlled blxirring of the image so that a point of light will 
always cover several color sensors; however, this anti-alias blurring has the disadvantage 
5 of reducing image sharpness. 

Another form of data compression of colored images adds the three primary colors 
to form a black and white image, called a Y channel image, that is stored at full resolution. 
Red and blue channel images are then individually differenced with this Y channel image. 
Typically the red channel minus the Y channel is called the U channel and the blue channel 
10 minus the Y channel is called the V channel. The U and V chaimel images are stored at 
lower resolutions than the Y channel image. In a specific implementation, alternating 
pixels store both Y and U records, and the next pixel stores both Y and V records, two 
numbers per pixel rather than three numbers needed to store all three primary colors. 
However, the disadvantage with using so called YUV color space is that 75% of the state 
15 space is wasted; that is, if Y, U, and V numbers were generated randomly within the full 
range appropriate to each, 75% of the generated numbers would produce invalid colors, or 
colors outside the range of real world colors. 

Single chip color area sensors, commonly used in almost alf consumer electronic 
imaging products, place color filters over each sensor in a single two dimensional array of 
20 sensors. Thus each pixel represents a single color, and this may be thought of as a data 
compression scheme in which two of the three primary colors are discarded for each pixel 
Several patterns of colors are available in the art, such as the Bayer array that assigns half 
the pixels to green in a checkerboard; the striped color array such as that used in a 
Polaroid digital camera in which entire columns cycle between red, green, and blue; and a 
25 technique commonly used in video cameras that uses cyan, magenta, yellow, and green in 
a repeating square. All of these techniques have been previously described above. 

It is, therefore, an object of this invention to provide an improved method and 
apparatus for scanning images with a variety of sensor arrangements. 

It is yet another object of the present invention to scan images so that the image 
30 data may be compressed for easier storage and manipulation. 

It is still another object of the present invention to provide an improved method 
and apparatus for scanning images which decreases the time it takes to scan the image. 

It is another object of the present invention to provide a method and apparatus for 
recovering a scanned color image. 
35 It is another object of the present invention to provide a method and apparatus for 

scanning images with inftared light in a single pass using a color filter matrix. 

It is another object of the present invention to provide a method and apparatus for 
scanning images with infrared light in a single pass using an existing color filter matrix. 
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It is another object of this invention to provide a method and apparatus for 
scanning images under visible and infrared light so that surface defects in the scanned 
images are reduced or eliminated. 

To achieve these and other objects which will become readily apparent upon 
5 reading the attached disclosure and appended claims, an improved method and apparatus 
for scanning an image is provided. Additional objects, advantages, and novel features of 
the invention will be set forth in part in the description which follows, and in part will 
become apparent to those skilled in the art upon examination of the following, or may be 
learned by practice of the invention. The objects and advantages of the invention may be 
10 realized and attained by means of the instrumentalities and combinations particularly 
pointed out in the appended claims. 

SUMMARY OF THE INVENTION 

The present invention provides an improved method and apparatus for scanning 
15 an image. A plurality of sensors is arranged in groups. The first group of sensors is 
behind a filter material selective to both a first color and infrared light A second group is 
behind a second filter material which is selective to a second different color and infrared 
light An image at a first scan time is illuminated with light functionally firee of infrared 
which is sensed with the first group of sensors to generate a first color image, and with the 
20 second group of sensors to generate a second color image. The image is then illuminated 
by a second light source containing infirared at a second scan time and sensed by at least 
one of the first or second group of sensors to generate an infrared image. From the first 
color image, the second color image, and the infrared image, a corrected color image is 
generated which is substantially free of media-based defects. 

25 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a perspective view of a prior art trilinear array component as used in a 
typical image scarming application; 

FIG. 2 is a magnified view of part of the trilinear array component shown in 

30 FIG. 1; 

FIG. 3 is a graphical representation of a trilinear array component; 
FIG. 4 is a graphical representation of the scanning of a single row of pixels shown 
in FIG. 3; 

FIG. 5 is a graph of visible and infrared light transmissivity; 
35 FIG. 6 is a graphical representation of a scan of a single row of pixels by a trilinear 

array component with alternating visible and infrared light sources; 

FIG. 7 is a graphical representation of a single row infrared scan at an increased 
resolution; 
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FIG. 8 illustrates an alternative representation of the information shown in FIG. 6 
and indicates the color component not present; 

FIG. 9 is a graphical representation of three pixels used in conjunction with a 
method for recovering missing color components; 
5 FIG. 1 0 illustrates an alternate embodiment of a trilinear array; 

FIG. 11 is a graphical representation of the color pattern generated by the trilinear 
array of FIG. 10; 

FIG. 12 is a graphical representation of a color pattern produced by a scanner in 
which no pixel lacks green; and 
10 FIG. 13 is a perspective view of an illuminating light source. 

DETAILED DESCRimON 

A suitable apparatus for practicing the present invention may be constructed by 
modifying the design of a typical single pass image scanner utilizing a trilinear array and a 

15 gas discharge tube that diffuses white light. About ten light emitting diodes ("LEDs") 
may be attached around the gas discharge tube in a pattern that does not directly interfere 
with the light path between the tube and the fihn substrate being illuminated. A suitable 
LED is an aluminum gallium arsenide LED manufactured by Cleairex Technology, Inc. of 
Piano, Texas. These LEDs may be pulsed in sync with every third scan of the scarmer by 

20 an external computer programmed to monitor the operation of. the scanner. Overall 
illumination of the substrate may need to be attenuated to avoid saturation. The 
attenuation may be accomplished by adding a 64% passing neutral density filter, such as a 
Kodak ND 0.3, to the optical path. After scanning the image, the processes described 
below are applied to the pixel data generated by the scanner to separate out the infrared 

25 channel, to reconstruct missing colors, and to apply surface defect correction. To reduce 
the amount of data to be processed, a simple data compression scheme may be used for 
color images that avoids the previously described inefficiencies of YUV color space while 
still providing less aliasing than the single color per pixel method common in color area 
array sensors. 

30 Considering this embodiment of the invention in more detail, FIG. 1 shows a 

typical trilinear array component 100 used in a single pass scanner. A long rectangular 
vwndow 102 in the component 100 exposes a line of sensor elements 104, typically laid in 
parallel with a CCD shift register (not shown) to receive the charges from the sensor 
elements 104 and shift those charges to an amplifier (not shown). The amplifier is 

35 typically etched into the same silicon die as component 100, which outputs the amplified 
signal through pin connectors 106 for further processing. In this type of scarmer 
application, light from a lamp 110, which may be a gas discharge tube, illuminates an 
image on a fihn substrate 112. A specific point 1 14 on this image is focused with lens 1 16 
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onto a specific sensor element 118 within line 104 of the trilinear array 100. Typically 
the film substrate 112 is moved perpendicularly to the direction of the line 104 (as shown" 
by the arrow) to complete a two dimensional scan of the image. 

FIG. 2 shows a magnified view of the trilinear array component 100 of FIG. 1 . As 

5 seen in FIG. 2, the rectangular window 102 contains three separate sensor lines 202, 204, 
and 206, each of which consists of many individual sensing elements 210. The sensor 
lines 202, 204, and 206 may each lie behind colored filters 222, 224 and 226, respectively, 
so that any illuminating light must pass through, and be affected by, these filters. Filter 
222 over sensor line 202 passes both red and infrared light, filter 224 over line 204 passes 

10 green and infrared light, and filter 226 over line 206 passes blue and infi^d Ught. The 
specific order and color of the colored filters 222, 224 and 226 can be rearranged as 
desired. 

In FIG. 3, the layout of the sensors in the trilinear array 100 of FIG. 2 is 
represented graphically. The sensor line 302 under the blue selecting filter, called the blue 

15 colunm, consists of individual pixel sensors 304. The spacing of these sensors 304 is 
referred to as the sensor row pitch 318, and is usually the same as the pixel column pitch 
320. Similarly, there is a green column 306 and a red column 308. The spacing of these 
columns is known as the sensor column pitch 322. Typically, the sensor column pitch 
322 is a power of two (such as 2, 4, 8, 16, 32, etc.) times the sensor row pitch 318 to 

20 allow easy reduction to scans at resolutions reduced fit)m the highest resolution by 
powers of two. In a typical application, the ratio of sensor column pitch 322 to sensor 
row pitch 318 is 32, although a ratio of 4 is used in FIG. 3 for simplicity. In the 
representative trilinear array of FIG. 3, a single sensor row 310 consists of a single red 
sensor 312, a single green sensor 314, and a single blue sensor 316, each separated by the 

25 sensor column pitch 322. In actual use, the image being scanned and the trilinear array 
move relative to each other in a direction perpendicular to the sensor lines. Individual 
scans of the image are made at increments of the pixel column pitch 320, thereby 
producing a scanned image with image pbcels spaced according to both the pixel column 
pitch 320 and the pixel row pitch 318. This process creates scanned image information in 

30 a two-dimensional array of pixels wherein any specific pixel may be referenced by a 
column and row designation. 

FIG, 4 illustrates the movement of and image data collection from a single sensor 
row 310 (originally shown in FIG. 3) of a trilinear array. For purposes of illustration, the 
vertical axis in FIG. 4 represents the passage of time during a scan performed by sensor 

35 row 310. At a first scan time 412. the red sensor 414 aligns with pixel colunm 0 of the 
image, the green sensor 416 aligns with pbcel column 4, and the blue sensor 418 aligns with 
pixel column 8. At the next scan time 420, each of these sensors 414, 416, and 418 has 
moved to the right a distance equal to the pixel column pitch 320. As is evident in FIG. 4, 
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pixel column 8 is the first column to be sensed by all three colored sensors. In order to 
complete the image scan, at some later point in time the trilinear array must reposition- 
over the image to allow scanning of columns 0 through 7 by the green and blue sensors 
416 and 418. The first few columns which are not scanned by all the sensors before 

5 repositioning the sensor array defme a distance referred to as the preamble 422. After the 
scanning process has repeated a fixed number of times to fill the preamble 422, each pixel 
column will have been sensed by all three colored sensors. Specifically, note that at pixel 
column 8, the red sensor 414 made a pass at time 8, the green sensor 416 made a pass at 
time 4, and the blue sensor 418 made a pass at time 0. In this way, a triUnear array can 

10 sense the red, green, and blue components of every pixel of an image even though the 
individual colored sensor columns do not lie directly on top of each other, 

FIG. 5 illustrates a characteristic of almost all colored filters to pass infi:ared light 
to the sensors 414, 416 or 418 of FIG. 4. In particular, the thin colored filters used in 
typical trilinear arrays pass not only the colored light for which they were designed to be 

15 specific, but also any infrared light present at the time of the scan. The graph in FIG. 5 
shows the transmissivity of light through blue, green and red filters versus the wavelengtli 
of the light. The blue filter transmits visible light at wavelengths between 400-500 
nanometers as shown by curve 502, and also transmits infi-ared light having wavelengths 
beyond 750 nanometers as shown at 504. Curve 506 shows that a green filter transmits 

20 visible light at wavelengths between 500-600 nanometers, and also transmits infrared light 
as evidenced at 508. Curve 510 illustrates that a red filter transmits visible light at 
vravelengths between 600-700 nanometers, and also transmits infi-ared light at 512. To 
generate a scan in color, normally the infrared light must be removed fi-om the illuminating 
light source. Alternatively, the light source may be fiinctionally free of infrared light. 

25 This occurs when the amount of infirared light in the light source is known, such as 10%, 
and so may be mathematically removed from the scanned pixel information. For purposes 
of this disclosure, a light source which is functionally fi-ee of infi-ared light includes 
complete or partial elimination of infrared light from the light source. If the illuminating 
light source contains only infrared light, then all three sensors will transmit it. Thus, the 

30 scanner becomes an infirared scanner in which each pixel is scanned three times, once with 
each sensor line of the trilinear array. 

FIG. 6 illustrates movement and data collection of a single sensor row 310 when 
the illuminating light source is alternately switched during the scan between a light source 
containing red, green, and blue (R, G, B), but no infrared light; and a light containing only 

35 mfrared (I) light Specifically, the visible light source is on for scans at times 0 and 1 (602 
and 604, respectively). The illumination source then switches to infixed light for the 
scan at time 2 at 606, and the cycle is repeated. As seen in FIG. 6, this results in each 
pixel between the preamble 608 and postamble 610 being sensed in infrared light and in 
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two of the three colors (red, green and blue). In particular, column 8 is sensed in infiared 
light and all colors except red, column 9 in infrared light and all colors except green, and " 
column 10 in infrared light and all colors except blue. This sequence repeats across the 
image. 

5 The scanning method shown m FIG. 6 uses a single pass scanner with a trilinear 

color sensor and a physically modified illuminating light source to generate a scan in which 
each pixel has an infrared record. This infrared record can be used to remove the effects of 
surface defects by diividing the visible records by the corresponding infrared records. 
More details on this method of surface defect correction may be found in U.S. Patent No. 

10 5,266,805 issued to the present inventor. Following surface defect correction, a relatively 
defect-free image remains in which each pixel is deficient in one of the three color 
components. However, these missing color components may be approximated from the 
adjacent color components which were detected by the correspondmg color sensor. 

The missing color components may potentially lower the quality of the scanned 

15 image. In practice, however, it has been found that negligible quality is actually lost if the 
missing color is estimated using methods disclosed below. This is because the important 
luminance resolution of a black and white image remains unaltered by discarding one of the 
three color components. The resolution reduction is confined entirely to the chrominance 
channel; however, as in television and JPEG compression, the color resolution can be 

20 significantly reduced relative to the luminance resolution. Discarding one of the three 
color components provides true data compression that reduces the data size of an image 
while potentially losing negligible quality. In the case of a scanner, this type of data 
compression permits the addition of infrared-responsive surface defect correction without 
mandating the scan time penalty of making a fourth scan. With the method of FIG. 6, the 

25 scanner proceeds at the same speed as in the non-infrared sensing method shown in FIG. 
4. 

In situations where no data loss can be tolerated, the pixel column pitch can be 
reduced to produce a higher resolution scan. An interesting case results when the pixel 
column pitch is reduced to 2/3 of the pixel row pitch, so the loss of one of the three color 

30 components at each pixel is compensated with 3/2 the number of pixels. After this higher 
resolution scan, the image can be sized down so the resized pixel column pitch again 
matches the pixel row pitch. Of course, the increase in resolution does not need to be 3/2, 
but may be any convenient number. FIG. 7 illustrates the case where the resolution is 
doubled. In fact, it caimot normally be increased to 3/2. This may be explained as 

35 follows. If the ratio of the sensor column pitch 702 to the pixel column pitch 704 is "N," 
then in order for individual scans to align, "N" must be an integer. Furthermore, in order 
to distribute the infrared pixel information evenly, "N" must not be a multiple of three. If 
"N" were simply multiplied by 3/2 as a result of increasing resolution, then if "N" were 



-9- 



wo 98/34397 



PCT/US98y01741 



still an integer, the product would have three as a factor. In practice, when "N" is a 
multiple of three, the resulting scan produces a cycle of two columns with red, green, and - 
blue but no infrared pixel information, followed by a column with infrared information 
sensed three times. In the case illustrated in FIG. 6 where "N" is 4, "N" could be 
5 increased to 5, 7, or even 8 (as illustrated in FIG. 7) but not to 6 or 9 if this problem is to 
be avoided. 

FIG. 8 illustrates the topology of FIG. 6 in an alternative representation. The 
missing red color component in pixel colunm 8 of FIG. 6 is shown as ("R bar" or "not R") 
at 802. The missing colors in the other rows are analogously shown under a bar. FIG. 9 

10 portrays a method for decompressing pixel data to recover missing color information 
when the illuminating light source is switched between visible and infrared light during 
image scanning (as previously described in conjunction with FIG. 6). For purposes of 
illusti-ation, the method is described for recovering green from a pixel missing green 902, 
which lies between a pixel missing red 904 and a pixel missing blue 906. Missing pbcel 

15 color information is indicated in FIG. 9 by the first letter of the missing color in 
parentheses. This general method applies to any color combination by substituting the 
corresponding names for the missing colors. The simplest method of estimating the 
missing green pixel value G2 of the center pixel 902 is to average the green pixel values of 
the immediately adjacent pixels 904 and 906. 

20 An alternative embodiment for recovering the missing green pbcel value of center 

pbcel 902 is to use the known color pixel values of the surrounding pixels and their 
corresponding rates of change. Most images are primarily monochromatic, so red, green, 
and blue colors within a small region of the image tend to change with similar degree. In 
the example shown in FIG. 9, the blue values Bl 908 and B2 910 for the left and center 

25 pixels are known. The difference B2-B1 specifies how fast detail in the image is causing 
blue to change when moving from the left pixel 904 to the center pixel 902. If it is 
assumed that the color green is changing across these same two pixels at about the same 
speed as the color blue changes, then the value of the green estimate of G2 912 originally 
obtained by averaging the green values of the adjacent pixels may be improved by adding 

30 the change in blue to Gl 914, the left pbcel, to give a new value of green for the center 
pixel G2B. Similarly, the estimate of G2 912 may be improved by using the change in red 
between the right pbcel R3 916 and the center pixel R2 918 to give an estimate of the 
change in green G2R based on the change in red. Averaging the two new estimates G2B 
and G2R gives an estimate for G2 912 that is theoretically perfect for a monochrome 

35 image, and very good for real world images. 

Additional variations on the data compression method described above are 
possible. For example, the difference in pbcel color values may be calculated in the 
logarithmic domam which has Uie effect of multiplying Gl 914 by the ratio of B2 910 
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over Bl 908 to produce the estimate G2B. This causes the estimate of green for the 
center pixel 902 to track the percentage change of blue when moving from the left pixel 
904 to the center pixel 902 rather than using just the absolute difference. This variation 
gives a good prediction for any image that is constantly colored, but may introduce noise 

5 in the missing primary of saturated colors. Yet another mathematical variation on the 
method uses the square root domain. The square root domain provides a compromise 
between the linear and logarithmic cases, and most images are stored in computer memory 
in this mode. The use of the word "subtraction" in the previously described methods is 
generally meant to include all such differencing operations described above. A further 

10 refinement of the method of FIG. 9 acknowledges that the colors, on average, do not all 
change exactly the same. This is done by multiplying the change in blue B2-B1 and in red 
R2-R3 by a factor close to, but less than unity, for example 0.9. The value for the factor 
can be calculated based on the correlation between the colors in an adjacent region around 
the missing color element. 

15 Often in a data compression method, it is not desirable to treat all pixel data in a 

given column of an image exactly alike. FIG. 10 shows an alternate topology of a trilinear 
array in which pixels of the even sensor rows, such as row 2 at 1002, are offset by one 
pixel spacing from the odd sensor rows, such as row 3 at 1004. This array produces the 
missing color pattern illustrated in FIG. 1 1 . Recovery of the missing colors could proceed 

20 using any of the methods described in conjunction with FIG. 9. Furthermore, recovery 
could also use the differences from pixels above and below, rather than just to the left and 
right of each pixel, because the pixels above and below are missing a color different from 
the one being calculated. For example, to estimate the missing green information of pixel 
1 102, one can add the change in blue from pbcel 1 104 to 1 102 to the green of pixel 1 104 to 

25 give a top green estimate; add the change in blue from pixel 1 1 06 to pixel 1 1 02 to the green 
of pixel 1 106 to provide a bottom green estimate, and average these with the left and right 
estimates as previously described to give an improved estimate of the missing green data 
of pixel 1102. 

Other trilinear array topologies are possible. For example, in the sensor of FIG. 

30 10, row 0 and 1 can remain as shown, but row 2 may be offset two pixel columns to the 
right of row 0. This pattern may be repeated so row 3 is again aligned with row 0, row 4 
is offset one pixel to the right, and row 5 is offset two pbcels to the right This produces a 
sensor pattern in which the missing color of the pixel directly above a given pixel is 
different than the missing color of the pixel directly below that given pixel. Still other 

35 sensor offsets are possible with the intent of more randomly dispersing the missing colors 
throughout the image. Such an offset helps reduce or prevent a sequence of vertical lines 
in the scanned image vAnch would be likely to create a moire effect. 
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FIG. 12 portrays a pattern of missing pixels in which no pixel lacks green, but in 
which red and blue are each missing throughout the scanned image in a complementary ' 
checkerboard pattern. All color differences are calculated relative to green in estimating 
the red and blue missing colors. Such a green priority pattern provides the data 

5 compression described herein with less image quality loss. 

In general, the reconstruction of missing colors can be performed on the image as it 
is scanned. Then, after reconstruction of each color, including the reconstructed color, is 
divided by the associated infrared information to perform surface defect correction. 
Alternately, each color can be divided by the associated infrared data to perform surface 

10 defect correction as soon as it is scanned, and then after surface defect correction the 
reconstruction of missing colors performed on the defect corrected scanned colors. 

As previously described, there is a need for illuminating light sources which are 
switched between infrared and visible in order to practice this invention. Typical 
switchable visible light sources include a pure gas discharge tube, such as xenon, or light 

15 emitting diodes. In a more typical scenario the illuminating light contains a visible source 
that cannot be rapidly switched. Visible light sources such as incandescent lights glow 
with heat for several milliseconds after the power is removed. Visible light sources such 
as mercury vapor discharge tubes glow for several milliseconds after the power is removed 
because of the phosphor used to convert the ultraviolet light of mercury into visible light. 

20 Thus, even though the ultraviolet and mercury frequency band components turn off 
rapidly, the phosphor continues to glow with visible light for some time. 

FIG. 13 illustrates an illuminating light source 1300 consisting of a mercury 
discharge tube 1302 surrounded by infrared light emitting diodes 1304. The tube power 
source/controller 1306 operates the discharge tube 1302 while the LED power 

25 source/controller 1308 manages the operation of the LEDs. By surrounding the tube 1302 
with the diode light sources 1304, the milky white phosphor in the tube tends to diffuse 
the infrared light emitted by the diodes 1304 so the infrared and visible light appear to 
radiate from the same directions. This uniformity in source light emission is important to 
accurate surface defect correction. The illuminator can be driven by leaving the visible 

30 light source 1302 on for all scans and switching the infrared source 1304 on for every third 
scan, as has been previously described herein. When this is done, the infrared record will 
also contain visible light data corresponding to the "missing" color because that is the 
color of the filter over the sensor making the infi-ared scan. The overall illumination must 
be adjusted so the visible light plus the infrared light will not saturate the sensor. To 

35 reduce the amount of visible light contamination, the sensitivity of the array may be 
decreased. This can be accomplished by shortening the integration time and/or by 
increasing the infrared illumination. 
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When using the illuminator 1300 disclosed in FIG. 13 with the visible light source 
1302 constantly on, the following steps will ensure proper scanning and surface defect 
correction. First, the "missing" color for each pixel is calculated from the adjacent pixels 
as before. The estimated value for that color is then subtracted from the sensed value 

5 which includes measurements of the "missing" color in addition to the infrared 
information. After the subtraction, this yields recovered infrared-only information for the 
pixel. The final step in the process is to divide the three colors for each pixel, including 
the two measured colors and one estimated color, by the recovered infiiared information to 
perform surface defect correction. 

10 In an alternate embodiment, the gas discharge lamp 1302 can be switched off 

during the infrared scan. This would attenuate, but not eliminate, the contamination of the 
infrared measurement by visible light. The mercury bands and different phosphor 
components each have different persistence, so each of the three color components red, 
green, and blue, decays to a different degree. Typically blue light will decay fastest, and 

15 hence exhibit the least contamination, while red light decays the slowest. When using 
such a switched illuminator, the estimated value for the "missing" color is calculated as 
previously explained; however, before being subtracted from the infrared record, it is first 
multiplied by a constant known by measurement to represent the amount of that color 
remaining after the visible light source is extinguished. Typical values for the constant 

20 would be 0.2 for blue, 0.5 for green, and 0.6 for red. By switching the visible light source 
off during the infrared scan, there is less contamination, and therefore less to be 
subtracted. This also helps to reduce noise and saturation problems. 

Although the present embodiment has been described with a fluorescent visible 
source and an LED infrared source, there are many choices for visible and infrared sources. 

25 An alternate embodiment contemplates an unswitched incandescent lamp behind a 
rotating filter wheel that removes infrared with a filter for two scans, and passes only 
infrared with a second filter for the third scan. In this embodiment, the filter wheel must 
be synchronized with the sensor. In another version, a visible fluorescent source can be 
supplemented with an incandescent source behind a filter passing only infrared light, and 

30 further behind a rotating wheel or shutter that passes the infrared light only for the 
infrared scan. There are many combinations of visible sources, infrared sources, filter 
wheels and shutters, and these are meant only to illustrate some of the many ways to 
provide the required light source in which infixed light is switched on alternating with, or 
in addition to, visible light 

35 The present invention has been described with a trilinear CCD array having a 

pulsed infrared source to generate the infi:ared image. However, high speed scanners such 
as the Tamron Fotvix utilize an area anay to avoid the time required for mechanical 
movement of a linear array. These area array scanners typically employ a matrix of tiny 
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color filters over each element of the area array to sense the color image. Typically, all of 
the filters pass infiared light Because the image presented to the sensor array of the 
present embodiment must be free of infirared light, the infi:ared component is removed by 
a filter in the light path. To use a smgle chip area array, two scans are performed: a 

5 conventional visible scan containing no infi-ared light, and a second infrared scan containing 
infrared light Because all colored filters typically pass infrared light, the second scaimed 
image has an infrared value for each pixel of the image. By dividing the visible values 
sensed for each pixel by the infrared values sensed for each pixel, surface defect correction 
may be practiced. The smface defect corrected image under the color matrix is then 

10 processed conventionally to recover the frill color image. 

In another embodiment using an area array, the visible light source is extinguished 
for the infrared scan, and the infrared light source is extinguished for the visible scan. This 
may be done with two light sources which are alternately excited. The infrared source can 
be a group of infrared LEDs, and the visible source can be a xenon discharge lamp although 

15 many other sources known in the art can be substituted to produce visible and infrared 
light. Alternatively, the light sources could be placed behind shutters to obviate switching 
each source itself. In still another alternate embodiment, a single source containing both 
visible and infrared light such as an incandescent lamp could be placed behind a filter 
wheel to alternately pass only visible light and only infrared light. In such an 

20 embodiment, the visible value sensed for each pixel is then divided by the infrared value 
sensed for that pixel. Finally, the resulting color matrix is decoded into frill color. 

In an alternate version of the present invention using an area array, the visible light 
is not extinguished during the infrared scan so the infrared scan also contains some visible 
light. The visible light contained in the value sensed for each pixel during the infrared scan 

25 is canceled by subtracting the visible value sensed for that pixel in the visible scan to leave 
a value representing pure infrared. By dividing the visible value by the pure infrared 
value, surface defect correction may be implemented. 

While this invention has been described with an emphasis upon certain preferred 
embodiments, variations in the preferred composition and method may be used and the 

30 embodiments may be practiced otherwise than as specifically described herein. 
Accordingly, the invention as defined by the following claims includes all modifications 
encompassed within the spirit and scope thereof 
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CLAIMS 

1 . A method for scanning an image on a substrate containing defects which uses a 
plurality of sensors arranged in groups, wherein a first group is behind a first filter 
5 material selective to a first color of light and infrared light, and a second group is behind a 
second filter material selective to a second different color of light and infirared light, said 
method comprising: 

illimiinating the image at a first scan time with a first light source functionally fiiee 
of infrared light; 

10 sensing light of the first color with the first group of sensors to generate a first 

color image, and light of the second color with the second group of sensors to generate a 
second color image; 

illuminating the image at a second scan time with a second light source containing 
infixed light; 

15 sensing infrared light with at least one of the first or second group of sensors to 

generate an infixed image; and 

generating from the first color image, second color image, and infrared image, a 
corrected color image substantially free of the defects. 

20 2. The method of claim 1 wherein the first group and second group of sensors are 
arranged in first and second parallel rows separated by an offset. 

3. The method of claim 2 wherein the first scan time is one of a first sequence of scan 
times with light functionally free of infrared light and the second scan time is one of a 

25 second sequence of scan times in which the image is illuminated with a light source 
containing infrared light. 

4. The method of claim 1 wherein the first group of sensors is arranged in a two 
dimensional grid. 

30 

5. The method of claim 1 wherein the first color is red, the second color is green, and 
further comprising a third group of sensors behind a third filter material selective to blue 
light and infixed light. 

35 6. The method of claim 3 wherein the first sequence of scan times includes a first 
series of individual scan times spaced between a second series of individual scan times of 
the second sequence. 
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7. The method of claim 6 wherein the first and second series of individual scan times 
each comprises two individual scan times. 

8. The method of claim 6 wherein the first and second series of individual scan times 
5 each comprises three individual scan times. 

9. The method of claim 8 further comprising moving the parallel rows of sensors by 
an incremental amount in a direction perpendicular to the parallel rows between the 
individual scan times of the first sequence. 

10 

10. The method of claim 6 further comprising moving the parallel rows of sensors by 
an incremental amount in a direction perpendicular to the parallel rows between the 
individual scan times of the first and second sequences. 

15 11. The method of claim 10 fiirther comprising conducting individual scans of the 
second sequence based on the incremental amount so as to spatially overlay an mdividual 
scan of the first sequence. 

12. The method of claim 11 wherein an individual scan of the second sequence 
20 overlays two individual scans of the first sequence. 

1 3 . The method of claim 1 2 further comprising 

conducting a first group of individual scans selective to infrared, green, and blue 
light at a first pixel row site; 
25 conducting a second group of individual scans selective to infi-ared, red, and blue 

light at a second pixel row site adjacent to the first pixel row site; and 

conducting a third group of individual scans selective to infrared, red, and green 
light at a third pbcel row site adjacent to the second pixel row site. 

30 14. The method of claim 13 wherein a value for green is calculated for a pixel at the 
second pixel row site. 

1 5. The method of claim 1 3 wherein a value for red is calculated for a pixel at the first 
pixel row site, a value for green is calculated for a pixel at the second pixel row site, and a 
35 value for blue is calculated for a pixel at the third pixel row site. 
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16. The method of claim 14 wherein the calculation for a value of green in the second 
pixel row site includes the step of averaging the value for green in adjacent pixels in the - 
first and third pixel row sites. 

17. The method of claim 16 wherein the calculation further comprises calculating the 
change of red value in adjacent pixels between the first and second pixel row sites. 

18. The method of claim 16 wherein the calculation further comprises calculating the 
change of red value in adjacent pixels between the first and second pixel row sites, and the 
change in blue value in adjacent pixels between the third and second pixel row sites. 

1 9. The method of claim 1 wherein the second light source comprises visible light. 

20. The method of claim 19 wherein the first light source is a first lamp emitting 
visible light, and the second light source emits a mixture of light fi-om the first lamp and 
light fix)m a switchable infrared lamp. 

21. The method of claim 20 wherein the switchable infrared lamp is a light emitting 
diode. 

22. The method of claim 20 wherein the light from the switchable infixed lamp is 
calculated as a difference between the second light source and the first light source. 

23. The method of claim 1 wherein the second light source substantially excludes light 
in the visible range. 

24. The method of claim 23 wherein the first light source is a first switchable lamp, 
and the second light source is a second switchable lamp. 

25. The method of claim 1 wherein the second light source emits light from a light 
emitting diode. 

26. A method for compressing data representing a color image consisting of pixels, 
wherein each of a set of pixels has a color specified by three primary color components, 
the method comprising: 

voiding a first one of the primary color components of a first plurality of first 
color pixels; and 
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voiding a second one of the primary color components of a second plurality of 
second color pixels. 

27. The method of claim 26 wherein the plurality of first color pixels is arranged in a 
5 line. 

28. The method of claim 26 wherein the plurality of first color pixels is arranged in a 
checkerboard, 

10 29. A method of decompressing a color image comprising: 

receiving a color image consisting of pixels, wherein each of a set of pixels has a 
color specified by three primary color components; 

voiding a first one of the primary color components for a first plurality of first 
color pixels; and 

1 5 calculating the voided color components for the fu:st color pixels. 

30. The method of claim 29 wherein a value for one of the primary color components 
is calculated for a pixel at a second pixel row site. 

20 31. The method of claim 29 wherein the calculation for a value of one of the primary 
color components in the second pixel row site includes the step of averaging the value for 
that primary color component in adjacent pixels in the first and third pixel row sites. 

32. The method of claim 30 wherein the calculation further comprises calculating the 
25 change of red value in adjacent pixels between the first and second pixel row sites. 

33. An apparatus for scanning an image on a substrate containing defects comprising: 

a plurality of sensors arranged in groups, wherein a first group is behind a first 
filter material selective to a first color of light and infrared light, and a second group is 
30 behind a second filter material selective to a second different color of light and infirared 
light; 

means for illuminating the image at a first scan time with a first light source 
fimctionally free of infixed light; 

means for sensing light of the fu-st color with the first group of sensors to generate 
35 a first color image, and Hght of the second color with the second group of sensors to 
generate a second color image; 

means for illuminating the image at a second scan time with a second light source 
containing mfiared light; 
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means for sensing infrared light with at least one of the first or second group of 
sensors to generate an infrared image; and 

means for generating from the first color image, second color image, and infrared 
image, a corrected color image substantially free of the defects. 

5 

34. The apparatus of claim 33 wherein the first group and second group of sensors are 
arranged in first and second parallel rows separated by an offset. 

35. The apparatus of claim 34 wherem the first scan time is one of a first sequence of 
10 scan times with light fimctionally fi*ee of infirared light, and the second scan time is one of 

a second sequence of scan times in which the image is illuminated with a light source 
contaming infrared light. 

36. The apparatus of claim 33 wherein the first color is red, the second color is green, 
15 and further comprising a third group of sensors behind a third filter material selective to 

blue light and infrared light 

37. The apparatus of claim 35 wherein the first sequence of scan times includes a first 
series of individual scan times spaced between a second series of individual scan times of 

20 the second sequence. 

38. The apparatus of claim 37 wherein the first and second series of individual scan 
times each comprises two individual scan times. 

25 39. The apparatus of claim 37 wherein the first and second series of individual scan 
times each comprises three individual scan times. 

40. The apparatus of claim 33 wherein the parallel rows of sensors are moved by an 
incremental amount in a direction perpendicular to the parallel rows between the 

30 individual scan times of the first sequence. 

41. The apparatus of claim 33 wherein the parallel rows of sensors are moved by an 
incremental amount in a direction perpendicular to the parallel rows between the 
individual scan times of the first and second sequences. 

35 

42. The apparatus of claim 41 wherein the incremental amount enables, in conjunction 
with the offset between parallel rows of sensors, individual scans of the second sequence 
to spatially overlay individual scans of the first sequence. 
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43. The apparatus of claim 42 wherein the incremental amount enables an individual - 
scan of the second sequence to overlay two individual scans of the first sequence. 

5 44. The apparatus of claim 43 wherein sites of overlay are pixel row sites, and 
wherem a first pixel row site includes individual scans selective to infrared, green, and blue 
light; a second pbcel row site adjacent to the first pixel row site includes individual scans 
selective to infrared, red, and blue light; and a third pixel row site adjacent to the second 
pixel row site includes individual scans selective to infrared, red, and green light 

10 

45. The apparatus of claim 44 further comprising means for calculating a value for 
green for a pixel at the second pixel row site. 

46. The apparatus of claim 44 further comprising means for calculating a value for red 
15 for a pixel at the first pixel row site, a value for green for a pixel at the second pixel row 

site, and a value for blue for a pixel at the third pixel row site. 

47. The apparatus of claim 45 wherein the means for calculating a value of green in the 
second pixel row site further comprises means for averaging the value for green in adjacent 

20 pixels in the first and third pixel row sites. 

48. The apparatus of claim 47 wherein the means for calculating further comprises 
means for calculating the change of red value in adjacent pixels between the first and 
second pixel row sites. 

25 

49. The apparatus of claim 47 wherein the means for calculating further comprises 
means for calculating the change of red value in adjacent pixels between the first and 
second pixel row sites, and the change in blue value in adjacent pixels between the third 
and second pbcel row sites. 

30 

50. The apparatus of claim 33 wherem the second light source emits visible light. 

51. The apparatus of claim 50 wherein the first light source is a first lamp emitting 
visible light, and the second light source emits a mixture of light from the first lamp and 

35 light from a switchable infrared lamp. 

52. The apparatus of claim 51 wherein the switchable infrared lamp is a light emitting 
diode. 
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53. The apparatus of claim 55 wherein the light from the switchable infrared lamp is" 
calculated as a difference between the second light source and the first light source. 

5 54. The apparatus of claim 33 wherein the second light source substantially excludes 
light in the visible range. 

55. The apparatus of claim 54 wherein the first light source is a first switchable lamp, 
and the second light source is a second switchable lamp. 

10 

56. The apparatus of claim 33 wherein the second light source emits light from a light 
emitting diode. 

57. An apparatus for compressing data representing a color image consisting of pixels, 
15 wherein each of a set of pixels has a color specified by three primary color components, 

comprising: 

means for voiding a first one of the primary color components of a first group of 
first color pixels; and 

means for voiding a second one of the primary color components of a second 
20 group of second color pixels. 

58. The apparatus of claim 57 wherein the group of first color pixels is arranged in a 
line. 

25 59. The apparatus of claim 57 wherein the group of first color pixels is arranged in a 
checkerboard. 

60. An apparatus for decompressing a color image comprising: 

means for receiving a color image consisting of pixels, wherein each of a set of 
30 pixels has a color specified by three primary color components; 

means for voiding a first one of the primary color components for a first plurality 
of first color pixels; and 

means for calculating the voided color components for the first color pixels. 

35 61 . The apparatus of claim 60 further comprising means for calculating a value for one 
of the primary color components for a pixel at a second pixel row site. 
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62. The apparatus of claim 60 further comprising means for calculating a value for one 
of the primary color components in the second row site by averaging the value for that - 
primary color component in adjacent pixels in the first and third pixel row sites. 
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FIG. 10 
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